Skip to content

Karafka & Waterdrop: support different configurations per topic#5107

Draft
Drowze wants to merge 6 commits intoDataDog:masterfrom
Drowze:karafka-configuration-per-topic
Draft

Karafka & Waterdrop: support different configurations per topic#5107
Drowze wants to merge 6 commits intoDataDog:masterfrom
Drowze:karafka-configuration-per-topic

Conversation

@Drowze
Copy link
Contributor

@Drowze Drowze commented Dec 2, 2025

NOTE: this pull request is blocked until #4876 and #5120 are merged


What does this PR do?
Supports different karafka/waterdrop configurations per topic.

Motivation:
Most of our Karafka consumers simply iterate through a kafka batch of messages and process them individually. Some of our consumers however have a some logic to pre-process a batch (e.g.: split in smaller batches based on some logic) to process more efficiently later.
In such cases we want to selectively disable Datadog's Waterdrop/Karafka tracing and handle tracing manually ourselves.

Change log entry

Additional Notes:

How to test the change?

Automated tests provided.

@github-actions github-actions bot added integrations Involves tracing integrations tracing labels Dec 2, 2025
@Drowze Drowze force-pushed the karafka-configuration-per-topic branch 2 times, most recently from 0aee689 to 0faf894 Compare December 3, 2025 15:59
By the time that `Karafka::App#initialized!` is called,
`Karafka.producer` was already initialized.

On top of that, there's really nowhere we can hook in the Karafka
initialization where we can be sure that `Karafka.producer` wasn't yet
initiaized - that's because the Karafka initialization is NOT
necessarily tied to the WaterDrop initialization. For instance, the
following is a possible scenario:

```
$producer = WaterDrop::Producer.new { ... }

Datadog.configure do |c|
  c.tracing.instrument :karafka
end

 # note that the producer was initialized before the Karafka app (and in
 # this case, even before datadog was configured)
Karafka::App.setup do |c|
  c.producer = $producer
end
```

So instead of trying to hook somewhere before `Karafka.producer` is
initialized, let's simply listen to a Karafka after-initialization event
and append our middleware to the producer when that event is triggered.
@vpellan
Copy link
Contributor

vpellan commented Jan 7, 2026

Hey @Drowze ! The two prerequisite PR have finally been merged, thanks for your patience !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community Was opened by a community member integrations Involves tracing integrations tracing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants